Ouch, Theo!

Slashdot, by way of kerneltrap, picked up a hilarious flamewar over on the OpenBSD mailing list. The short version is that some guy thinks virtualization involves code, and code has bugs, and therefore virtualization is only ever bad. Or something. I'll leave the histrionics to those who think there's a substantial argument lurking in there somewhere, but I just couldn't help myself when I saw the following appeal to authority:

While x86 hardware has the same page-protection hardware that an IBM 390 architecture machine has, modern PC machines are a mess. They are architecturally so dirty, that parts of the video, keyboard, and other IO devices are interfaced with even to do simple things like context switching processes and handling interrupts. Those of us who have experience with the gory bits of the x86 architecture can clearly say that we know what would be involved in virtualizing it, and if it was so simple, we would not still be fixing bugs in the exact same area in our operating system going on 12 years.

I'd like to call "BS" on the above claim of inscrutability. No, you do not dork around with, e.g., the video card, to switch contexts. You may indeed mess about with an I/O device in "handling interrupts," but only because, gee, there's an interrupt to handle. In my seven years of debugging every goofball OS written for the PC on our VMM, I haven't encountered a single task switch that wasn't basically some variation of disabling interrupts, taking a couple spin locks, switching stacks, and changing the page table pointer. You know, kind of like it is on every architecture on the planet.

Denier's Dilemma

Shankar Vendatam at the Washington Post writes:

The federal Centers for Disease Control and Prevention recently issued a flier to combat myths about the flu vaccine. It recited various commonly held views and labeled them either "true" or "false." Among those identified as false were statements such as "The side effects are worse than the flu" and "Only older people need flu vaccine."

When University of Michigan social psychologist Norbert Schwarz had volunteers read the CDC flier, however, he found that within 30 minutes, older people misremembered 28 percent of the false statements as true. Three days later, they remembered 40 percent of the myths as factual.

...

(By way of the often excellent, and often nutty, Overcoming Bias.) I'm reminded of the difficulty I've had explaining that VMs and security don't really have much to do with one another. Is it possible that, in attempting to explain something really convoluted to a crowd that barely cares, we sometimes do more harm than good? If you find yourself in a situation where you're in possession of a counter-intuitive result, that's really hard to explain, but important none the less, what's the ethical thing to do, given that saying "X" will register in a lot of folks' minds as "not X"?

Presenting VProbes

VMworld 2007 is coming up, and I'm more excited than usual this time around. Alex Mirgorodskiy, Robert Benson and I will be demonstrating a piece of technology called VProbes.

VProbes was born from an experience that many of you have probably shared: I was wondering why my computer was so slow. I have whiled away many hours over the years investigating this question, and I have rarely gotten a solid answer. Sometimes the slowness is intermittent, and by the time I've developed a theory, it has already disappeared. Other times, the diagnostic tool I want to run (top, or strace, or ltrace, or what have you) isn't installed in the VM, because it's a purpose-built VM with a constrained set of user-land tools. Or, the tools are there, but the system is not in a state to allow me to use them; e.g., it's hung early in boot, and I can't log in to the system. Still other times, I never really scratch the surface of the problem because it's Windows and I don't know my way around.

VProbes attempts to provide a set of tools for answering the question, "What the heck is this computer doing?" It's an open-ended question, so vprobes is accordingly open-ended, as well. In its current form, it provides an interactive, safe way of instrumenting a running VM at any level: from user-level processes down to the kernel, and even into VMware's VMM and hypervisor, if need be.

We also attempt to solve the "unfamiliar tools" problem, by papering over (to the degree possible) the differences among operating systems. There is a broad consensus among Windows and the UNIX flavors about the abstractions that operating systems provide: all major operating systems provide threads, a tree of named processes, named files in a hierarchical namespace with byte contents, sockets, etc. VProbes tries to get by on enough guest-specific knowledge to "wrap" those differences. Thus, for Windows, Linux, and any other operating system that VProbes can be "taught" to understand, the following vprobe script provides a rough and ready approximation of the top UNIX utility:

Guest_TimerIRQ ticks[curprocname()] <- 1;


"Big deal," you might be thinking. "I had top already." Well, you didn't have top on Windows. And you didn't have it while the machine was booting, or shutting down. And you certainly didn't have it if the machine in question was a virtual appliance that doesn't even allow logins.

Anyway, if this sounds interesting to you, please consider attending our VMworld breakout session on the 13th, from 3:30 to 4:30, numbered TA70. I should warn that this is basically a research prototype: we haven't committed to putting VProbes in any upcoming VMware product, and indeed, might never ship it at all. We're putting what we've got in front of the public now because we are at the point where we need feedback from real users to improve VProbes.

In the "credit where it's due" department: we owe an enormous debt in our thinking about this problem to our colleagues at Sun. I've never hidden my admiration for DTrace. We hope to improve on it. First, we are aiming to provide a Dtrace-like tool for other commercially important operating systems than Solaris. Second, VProbes can combine with other virtualization-based techniques in powerful ways. For example, VProbes and deterministic replay combine to make the most potent tool that I'm aware of for debugging intermittent performance anomalies. I'll leave applications to VMotion, snapshots, etc., as tantalizing hints...

Hmm. That's not what I think I wrote...

I'm perplexed by this peculiar interpretation of our HotOS paper:

Security has been touted as one of the benign by-products of virtualisation – but according to a recent study, that’s no longer the case.

Huh? "Security" is a big concept, covering many possible applications. Our paper is just a tiny footnote in a complicated discusssion about the role of VMMs in providing security; in fact, to the extent we challenge conventional wisdom at all, we suggest that the purported security threat posed by VMM-based rootkits is non-existent. To get out the sock puppets: VMMs are always detectable, which is a good thing.

I think I see where the author got a bit tripped up reading our paper, though. Currently, there is a trend for malware to disable itself in the presence of VMMs. We argue that this trend cannot continue, not because VMMs are becoming undetectable, but because VMMs are becoming too ubiquitous for malware authors to ignore. Note that this current fad among malware authors of refusing to do dirty business in a VM is not an inherent security benefit of VMs!

A nice thought experiment when thinking about security applications of virtual machines is to replace "VM" with "laptop." Suppose, for the sake of argument, security researchers started building honeynets out of laptops, because it was more economical to do so. For a while, malware authors might decide to detect that they're running on a laptop, and refuse to do so, in order to thwart the security researchers. (Note, of course, that it's completely trivial for user-level software to figure out what sort of hardware platform it's running on; this information is intentionally, usefully exposed by, e.g., /proc nodes in Linux, or the \Device directory on NT, etc.). The answer to this situation is not to attempt to cloak the laptop-iness of the hardware platform, though; attempting to do so is a bottomless pit of wasted effort. Instead, we simply observe that there are a lot of laptops out there, and figure that, in the long run, malware that refuses to run on laptops will be giving up too many targets to represent a Nash equilibrium for the black hats.

Honestly, though, I'm not that happy with the paper. We're telling a complicated story, and I'm not sure we tell it clearly enough. Even if Manek Dubash's misinterpretation were intentional*, the fact that it the paper can even be plausibly distorted to read the way he suggests means that, at some basic level, we've failed. Good technical writing is hard.

* Purely speculating, here. I'm not casting aspersions on anyone's intentions; an honest misunderstanding is, sadly, possible.

Overcoming bias

Not strictly virtualization-related, but overcoming bias is one of the more consistently interesting cross-disciplinary blogs I've come across. The same sorts of cognitive biases that cause people to make bad decisions about their happiness, finances, political preferences, etc., often cause errors in software engineering. Pawing through Wikipedia's list of cognitive biases, I can associate almost every single bias with one (or more) mistakes I've made in my professional life. Optimism bias? Zounds, what engineer isn't guilty of it on a daily bias? Information bias has often led me to waste hours in debugging collecting useless trivia. Some have even entered the lore of computing; ingroup bias is nothing but "NIH syndrome."

While "overcoming bias" is certainly a laudable goal in many professional fields, I wonder about its tractability. These biases are clearly hardwired, and I suspect no amount of mindfulness will prevent us from committing them. Getting rid of one of these well-attested biases would probably require the neurological equivalent of, e.g., making humans able to digest grass, or giving a human the VO2Max of a horse. Oh well; if nothing else, at least being aware of our limitations as thinking machines might prevent us from trying anything too impossibly stupid...

BluePill detection in two easy steps.

In our HotOS submission, we alluded to a large number of theoretically possible methods of detecting the presence of a hypervisor. I'm not sure which of these many ways the well-publicised BluePill challengers have chosen, but I thought I'd make things a bit more concrete.

In our paper, we outline family of detection methods based on resource consumption. In short, the hypervisor must use cache, memory bandwidth, TLB entries, etc., in the course of multiplexing the CPU. A guest can be made intentionally sensitive to these resources in order to detect a hypervisor. For instance, you can obtain a good estimate of the effective size of the hardware TLB with psuedo-code like the following:

PPN oldPhysPage, newPhysPage = Alloc...();
VA oldVirtAddr = MapSomewhere(oldPhysPage);
VA newVirtAddr = MapSomewhere(newPhysPage);
memset(oldVirtAddr, 0x11, PAGE_SIZE);
memset(newVirtAddr, 0x22, PAGE_SIZE);
PTE=base of hardware page table;
for (i = 0; i < BIGNUM; i++) {
   PTE[i] = MAKE_PTE(oldPhysPage);  // map old page
   (void)*(volatile char*)(i * PAGE_SIZE); // bring it into the TLB
}
/*
* TLB now contains between 0 and BIGNUM old mappings.
* Now modify the PTEs *without* flushing the TLB,
* and wait for the hardware to leak an entry
*/
for (i = 0; i < BIGNUM; i++) {
   PTE[i] = MAKE_PTE(newPhysPage);  // map old page
   if ((*(volatile char*)(i * PAGE_SIZE)) == 0x22) {
      printf("apparent tlb size: %d\n", i);
      break;
   }
}

OK, so you can size the TLB. So what? Well, just run the above loop twice, but the second time, stick an instruction that's guaranteed to trap out to the hypervisor (e.g., CPUID) before the loop. On bare hardware, this inocuous instruction will have no impact on the TLBs. In the presence of a hypervisor, the instruction will either flush the TLB entirely (on current Intel VT implementations), or, if you're on a clever SVM hypervisor that uses the ASID facility to avoid TLB flushes on every hypervisor invocation, at least 1 fewer TLB entry. (Every hypervisor must at least read the VMCB's exit code, requiring 1 data TLB entry. For increased sensitivity, you could also detect the VMM's code footprint by rejiggering the above to measure ITLB entries, an exercise left to the reader. Hint: the return instruction is a single byte on x86.)

This detection technique uses only TLB capacity, but a more sophisticated approach could easily include the hardware eviction policy. We could also apply these same technique to instruction and data caches. (Even the x86's coherent data caches can be sized using the INVD instruction, which discards cache contents without writing back dirty data.) A less naive approach that uses the hardware's event-based performance monitoring support (RDPMC and friends) would have an even richer menu of options for noticing virtualization-induced perturbations.

I've seen zero evidence that Rutkowska has pondered resource-based detection attempts like this, or indeed, any attacks more sophisticated than a "go-slow" loop between reads of the PIT. It is hard for me to imagine a "hypervisor" worthy of the name that doesn't leave noticable traces in resource usage. In fact, to the degree that a hypervisor goes to heroic lengths to prevent such a detection technique, e.g., by running a hardware-accurate cache simulator on every guest memory access, it will only open up wider timing discrepancies for the guest's HV-detector to exploit.

I can only conclude that in 2006 Rutkowska was either naive about the possibilities of such attacks, or that she consciously chose to make an outrageous and indefensible claim ("undetectable malware!!!!111") in order to draw attention to herself and her company. Given the peripheral evidence of Rutkowska's competence, I think the evidence favors the latter, but I'd simply love to hear from her on the subject...

BluePill hype going the way of all flesh

I'm not alone in my skepticism about BluePill's claims of undetectability. Let's pop some popcorn, folks! This could get good...

Please indulge me...

I've tried to be fairly disciplined about keeping this blog technical. Never the less, I would ask for your indulgence to announce the birth of Emmett Joseph Adams on February 27th, 2007 at 11:48am.

The little dude is three amazing weeks old now. Everybody is as healthy and happy as can be expected. A few observations from very, very early parenthood:

1. I'm amazed in the photo below how bloody peppy I look. And that was after being awake all night! I've already come to regard bloodshot eyes, 5 o'clock shadow, and coffee stains on clothes I've slept in as the norm. When I ask other parents when I can expect that to change, they invariably say something smarmy, like, "oh, in about eighteen years."
   
2. When I play guitar and sing for him, he kind of zones out for a while and then starts bouncing his legs and arms in time while making these little happy grunts.
3. Emmett's mother, Jean, is the most amazing person in the world.

OK, back to our regularly scheduled computer stuff!

Microsoft swallows the BluePill

Why doesn't Microsoft want users to run Vista in a VM? Why, because the presence of a VMM could mean someone has l33tly r00t3d your b0x0r! So nice of Microsoft to protect us shivering, ignorant, childlike customers from choosing how we'd like to run Windows. Ahh, the bliss of having all my decisions made for me. Thank you. I wonder if I should be just as worried about the inherent insecurity of running on a hypervisor once Longhorn ships? Oops, I almost forgot: best not to think of these things! Better to let Microsoft do the worrying for me.

Now that the BluePill pseudo-sploit has been widely publicized for over a year now, I'm still aware of no actual attacks. This, despite the fact that SVM and VT hardware are now extremely common, and getting more so by the day. Why would that be?

Well, first of all, SVM and VT make possible nothing that was not already possible before; VMware's software-only products are an existence proof. The BluePill-istas don't claim that SVM/VT make new exploits possible per se; rather, the claim is that SVM/VT make it possible to cloak the presence of a VMM rootkit completely.

Allow me to go on record: this claim is pure fantasy. In practice, it is always possible to detect the presence of a VMM, via timing attacks. "Aha!", my hand-wringing BluePill interlocutor exclaims: "But the VMM lets us hook RDTSC and the PIT and the APIC timer, etc., so we can show the VM whatever time we want! Or whatever!" The details of getting such an approach right in practice are tough beyond belief. Once you start thinking it through, you quickly realize the deck is stacked in favor of the would-be VMM detector. As a practical example, VMware's software goes to extremely clever lengths in coordinating and manipulating the various virtual time sources just to get certain versions of the linux kernel to boot. It took some smart people years to really drive a stake through the heart of the "pester mingo" boot-time failure in SMP VMs. This was the result of a real, naturally occurring guest code that had nothing to do with VM detection; you can imagine that a dedicated attack on the VMM's virtual time sources could be many orders of magnitude more effective.

So, I'm claiming you can always find out that there's a VMM underneath you. Should that worry us, too? Since some members of the security community are using VMs to study malware, the arms race between malware creators and investigators has temporarily taken a turn wherein some malware refuses to run in a VM. This has led some to think of the detectability of a VMM as a deficiency in the virtualization layer.

I don't see it that way. As VMs become more commonplace, malware that refuses to run in VMs will be naturally selected out. We don't expect, e.g., a Dell laptop to cloak the version of Windows it's running, or which chipset or video card is present, or for that matter which version of Windows is running. It's perfectly legit for software to query these aspects of its execution environment, having an orderly way to enumerate and version them is a valuable thing. As VM proliferation progresses, I suspect that we will come to think of VMMs in the same way: as another facet of the stack of hardware and software on which you happen to be running, whose presence is neither surprising nor secret.